{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 22 零拷贝,mmap和sendfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| 概念/技术                   | 描述                                                         | 核心流程或实现要点                                           | 主要优势                                                     | 典型适用场景                                  | 局限与注意事项                                               |\n",
    "| --------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | --------------------------------------------- | ------------------------------------------------------------ |\n",
    "| 零拷贝（Zero-Copy）         | 通过系统调用与内存管理手段，尽量减少内核与用户空间之间的数据搬运次数，从而降低 CPU 负载、态切换频率。 | 利用 DMA、mmap、sendfile 等机制，让数据在内核空间内完成或少经过用户空间。 | 降低 CPU 占用、提高吞吐、减少上下文切换。                    | 高吞吐文件拷贝、静态资源分发、日志/备份同步。 | 并非真正“0”拷贝；不同操作系统对 API 支持存在差异。           |\n",
    "| DMA（Direct Memory Access） | 外设直接与内存交换数据的硬件能力，不依赖 CPU 逐字节搬运。    | 设备控制器通过 DMA 通道把数据写入/读出内核缓冲区，CPU 只负责发起与收尾。 | 让 CPU 在数据传输期间空出处理其他任务，提供基础的“零拷贝”前提。 | 所有块设备与网卡 I/O 场景。                   | 仍需内核缓冲区，后续是否进入用户态取决于上层实现。           |\n",
    "| 传统缓冲区拷贝              | 经典 `read → write` 模式，在用户态读取数据再写出。           | 磁盘→内核 buffer→用户 buffer→内核 buffer→目标设备。          | 实现简单、跨平台、API 成熟。                                 | 中小文件搬运、快速脚本、兼容性要求高的程序。  | 四次拷贝+两次态切换，占用大量 CPU 与内存带宽。               |\n",
    "| mmap 内存映射               | 将文件内容映射到用户虚拟地址空间，让用户态直接访问内核页。   | DMA 填充内核页 → `syscall.Mmap` 建立映射 → 用户直接读写映射区 → `Msync` 刷盘。 | 避免内核→用户拷贝，可按需访问、支持大文件。                  | 数据库/日志文件处理、随机读取超大文件。       | 平台相关（Windows 需改用等效 API）；必须 `Munmap`、`Msync`，易触发 SIGBUS。 |\n",
    "| sendfile 系统调用           | 在内核空间内直接把文件数据发送到 socket 缓冲区。             | 文件 DMA 入内核 buffer → `sendfile` 将内核 buffer 拷贝到 socket buffer → NIC DMA 发送。 | 减少一次用户态拷贝+系统调用，显著提升文件→网络吞吐。         | 静态文件服务器、CDN 边缘节点、日志收集传输。  | 仅适用于明文 TCP（TLS 需额外处理）；Linux、macOS 参数不同；`conn.File()` 要谨慎关闭。 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 传统缓冲区拷贝（基线方案）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **做了什么**：借助 `io.Copy` 在用户态循环读写文件，是最常见的“传统拷贝”实现。\n",
    "- **拷贝路径**：磁盘 → 内核缓冲区 → 用户缓冲区 → 内核缓冲区 → 磁盘/网络。\n",
    "- **适用场景**：代码简洁、跨平台通用，适合小文件和对性能要求不高的场景。\n",
    "- **局限**：CPU 需要执行两次内核/用户空间的数据搬运，会造成较多态切换与 Cache 污染。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### mmap 内存映射拷贝（减少一次拷贝）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**做了什么**：利用 `syscall.Mmap` 让用户空间直接映射内核页，避免中间缓冲区拷贝。\n",
    "\n",
    "**性能收益**：减少一次 CPU 拷贝和一次态切换，尤其对大文件读取/随机访问效果显著。\n",
    "\n",
    "**优势亮点**：\n",
    "- 可在虚拟地址空间中映射超大文件（> 物理内存）。\n",
    "- 支持按需访问，突破传统以块为单位的读写粒度。\n",
    "\n",
    "**注意事项**：内存映射是平台相关接口，Windows 需改用 `syscall.CreateFileMapping` 等；同时要记得 `Munmap` 与 `Msync`，避免内存泄漏或数据未刷盘。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### sendfile 系统调用（内核内传输文件→网络）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**做了什么**：服务器通过 `syscall.Sendfile` 将磁盘文件在内核态直接拷贝到 socket buffer，客户端仅负责读取。\n",
    "\n",
    "**性能收益**：相比传统 “read → write” 模式，减少一次用户态拷贝以及一次系统调用（`read`/`write` → `sendfile`），高吞吐文件分发场景可显著降本增效。\n",
    "\n",
    "**适用场景**：静态文件服务器、CDN 边缘节点、日志归档等大量文件→网络的传输链路。\n",
    "\n",
    "**局限及注意**：\n",
    "\n",
    "- 示例使用 Linux 语法；若在 macOS 运行，需要改用 `unix.Sendfile` 并调整参数。\n",
    "- 获取 `conn.File()` 会复制一份底层 FD；务必配套 `Close()` 避免资源泄漏。\n",
    "- `sendfile` 不会自动处理 TLS/加密通道，若需要 HTTPS 需借助 SSL offload 或回退到用户态。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 性能测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "goos: linux\n",
      "goarch: amd64\n",
      "pkg: example.com/tcpdemo/filecopy\n",
      "cpu: AMD Ryzen 7 7840HS with Radeon 780M Graphics   \n",
      "BenchmarkCopyWithIO-16          \t       3\t  55886784 ns/op\t 600.40 MB/s\n",
      "BenchmarkCopyWithMmap-16        \t       3\t  85455825 ns/op\t 392.65 MB/s\n",
      "BenchmarkCopyWithSendfile-16    \t       3\t  55835796 ns/op\t 600.95 MB/s\n",
      "PASS\n",
      "ok  \texample.com/tcpdemo/filecopy\t1.605s\n"
     ]
    }
   ],
   "source": [
    "! cd 22 && go test -bench=. -benchtime=3x -run=^$ ./..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "goos: linux\n",
      "goarch: amd64\n",
      "pkg: example.com/tcpdemo/filecopy\n",
      "cpu: AMD Ryzen 7 7840HS with Radeon 780M Graphics   \n",
      "BenchmarkCopyWithIO-16          \t       5\t  54864308 ns/op\t 611.59 MB/s\n",
      "BenchmarkCopyWithMmap-16        \t       5\t  85117171 ns/op\t 394.21 MB/s\n",
      "BenchmarkCopyWithSendfile-16    \t       5\t  56840912 ns/op\t 590.32 MB/s\n",
      "PASS\n",
      "ok  \texample.com/tcpdemo/filecopy\t1.980s\n"
     ]
    }
   ],
   "source": [
    "! cd 22 && go test -bench=. -benchtime=5x -run=^$ ./... -- \\\n",
    "  -filecopy.sizeMiB=128 \\\n",
    "  -filecopy.chunkKiB=2048"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "goos: linux\n",
      "goarch: amd64\n",
      "pkg: example.com/tcpdemo/filecopy\n",
      "cpu: AMD Ryzen 7 7840HS with Radeon 780M Graphics   \n",
      "BenchmarkCopyWithIO-16          \t       5\t  54989769 ns/op\t 610.19 MB/s\n",
      "BenchmarkCopyWithMmap-16        \t       5\t  80831576 ns/op\t 415.12 MB/s\n",
      "BenchmarkCopyWithSendfile-16    \t       5\t  50873204 ns/op\t 659.57 MB/s\n",
      "PASS\n",
      "ok  \texample.com/tcpdemo/filecopy\t1.934s\n"
     ]
    }
   ],
   "source": [
    "! cd 22 && go test -bench=. -benchtime=5x -run=^$ ./... -- \\\n",
    "  -filecopy.sizeMiB=16 \\\n",
    "  -filecopy.chunkKiB=512"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo \"hello sendfile\" > /tmp/sample.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! go run sendfile_server_linux.go :5678 /tmp/sample.txt\n",
    "\n",
    "[server] listening on :5678, serving /tmp/sample.txt\n",
    "[server] transfer complete, bytes=15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! go run sendfile_client.go 127.0.0.1:5678 /tmp/received.txt\n",
    "\n",
    "[client] received 15 bytes in 281.102µs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 23 顺序IO和随机IO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| 场景/技术                | 操作方式             | 性能特点                               | 优势                             | 局限                              | 适用建议                        |\n",
    "| ------------------------ | -------------------- | -------------------------------------- | -------------------------------- | --------------------------------- | ------------------------------- |\n",
    "| 顺序访问内存/磁盘        | 连续地址读写         | CPU/GPU 预取命中率高；磁盘读写头少移动 | 吞吐量最大；实现简单             | 对离散访问不友好                  | 大文件扫描、日志回放、串行处理  |\n",
    "| 随机访问内存（极端乱序） | 逐字节乱序访问       | 频繁缓存未命中导致延迟大               | 可直接按地址访问                 | 性能易劣化                        | 若需随机性，可分块或改善布局    |\n",
    "| 随机访问磁盘（ReadAt）   | 多次 `ReadAt`        | 机械盘寻道+旋转等待昂贵                | 精确定位                         | IOPS 上限低                       | 尽量避免，或切换到 SSD/更大块读 |\n",
    "| mmap 内存映射文件        | 映射虚拟地址直接访问 | 内核按页调度；少一次用户态拷贝         | 访问方式“像内存”；可处理超大文件 | 32 位地址空间有限；需处理缺页异常 | 大文件随机读、零拷贝场景        |\n",
    "| 全量载入内存后随机读     | `Read` 到大缓冲      | 内存命中率高；随机读极快               | 速度最快                         | 受限于物理内存；GC 压力           | 小文件、高频随机读              |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "准备生成 random_sample.data，大小：128M\n",
      "文件已生成：random_sample.data\n",
      "-rwxrwxrwx 1 ningliu ningliu 128M Oct  1  2025 random_sample.data\n"
     ]
    }
   ],
   "source": [
    "! cd ./23 && ./generate_random_file.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 顺序内存/磁盘读写实验（Go）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "顺序写内存：9 ms\n",
      "顺序读内存：2 ms\n",
      "顺序写磁盘：66 ms\n",
      "顺序读磁盘：54 ms\n"
     ]
    }
   ],
   "source": [
    "package main\n",
    "\n",
    "import (\n",
    "\t\"fmt\"\n",
    "\t\"math/rand\"\n",
    "\t\"os\"\n",
    "\t\"time\"\n",
    ")\n",
    "\n",
    "const (\n",
    "\tarraySize = 10_000_000          // 约 10 MB，用于内存实验\n",
    "\tfileName  = \"seq_io_test.data\"  // 顺序磁盘 IO 测试文件\n",
    ")\n",
    "\n",
    "func main() {\n",
    "\tarr := make([]byte, arraySize)\n",
    "\n",
    "\tmeasure(\"顺序写内存\", func() {\n",
    "\t\tfor i := range arr {\n",
    "\t\t\tarr[i] = byte(i)\n",
    "\t\t}\n",
    "\t})\n",
    "\n",
    "\tmeasure(\"顺序读内存\", func() {\n",
    "\t\tvar sum int\n",
    "\t\tfor _, b := range arr {\n",
    "\t\t\tsum += int(b)\n",
    "\t\t}\n",
    "\t\t_ = sum // 防止被编译器优化掉\n",
    "\t})\n",
    "\n",
    "\tmeasure(\"顺序写磁盘\", func() {\n",
    "\t\tf, err := os.Create(fileName)\n",
    "\t\tif err != nil {\n",
    "\t\t\tpanic(err)\n",
    "\t\t}\n",
    "\t\tdefer f.Close()\n",
    "\n",
    "\t\tif _, err = f.Write(arr); err != nil {\n",
    "\t\t\tpanic(err)\n",
    "\t\t}\n",
    "\n",
    "\t\t// 确保数据真正写出（去掉下面一行可观察 Page Cache 影响）\n",
    "\t\tif err = f.Sync(); err != nil {\n",
    "\t\t\tpanic(err)\n",
    "\t\t}\n",
    "\t})\n",
    "\n",
    "\tmeasure(\"顺序读磁盘\", func() {\n",
    "\t\tf, err := os.Open(fileName)\n",
    "\t\tif err != nil {\n",
    "\t\t\tpanic(err)\n",
    "\t\t}\n",
    "\t\tdefer f.Close()\n",
    "\n",
    "\t\tif _, err = f.Read(arr); err != nil {\n",
    "\t\t\tpanic(err)\n",
    "\t\t}\n",
    "\t})\n",
    "}\n",
    "\n",
    "// measure 负责统一计时打印\n",
    "func measure(name string, fn func()) {\n",
    "\tstart := time.Now()\n",
    "\tfn()\n",
    "\tfmt.Printf(\"%s：%d ms\\n\", name, time.Since(start).Milliseconds())\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 随机位置读取文件（ReadAt 方式）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ReadAt 随机读文件：8802 ms\n"
     ]
    }
   ],
   "source": [
    "package main\n",
    "\n",
    "import (\n",
    "\t\"fmt\"\n",
    "\t\"math/rand\"\n",
    "\t\"os\"\n",
    "\t\"time\"\n",
    ")\n",
    "\n",
    "const (\n",
    "\trandomFile       = \"./23/random_sample.data\" // 需提前准备的测试文件\n",
    "\trandomReadTimes  = 50000                // 随机读取次数\n",
    ")\n",
    "\n",
    "func main() {\n",
    "\t// 打开文件（只读）\n",
    "\tf, err := os.Open(randomFile)\n",
    "\tif err != nil {\n",
    "\t\tpanic(err)\n",
    "\t}\n",
    "\tdefer f.Close()\n",
    "\n",
    "\tstat, err := f.Stat()\n",
    "\tif err != nil {\n",
    "\t\tpanic(err)\n",
    "\t}\n",
    "\tsize := stat.Size()\n",
    "\tif size == 0 {\n",
    "\t\tpanic(\"文件大小为 0，无法测试\")\n",
    "\t}\n",
    "\n",
    "\tbuffer := make([]byte, 1) // 每次读取单字节，最不理想的访问模式\n",
    "\n",
    "\tmeasure(\"ReadAt 随机读文件\", func() {\n",
    "\t\tfor i := 0; i < randomReadTimes; i++ {\n",
    "\t\t\toffset := rand.Int63n(size)\n",
    "\t\t\tif _, err = f.ReadAt(buffer, offset); err != nil {\n",
    "\t\t\t\tpanic(err)\n",
    "\t\t\t}\n",
    "\t\t}\n",
    "\t})\n",
    "}\n",
    "\n",
    "func measure(name string, fn func()) {\n",
    "\tstart := time.Now()\n",
    "\tfn()\n",
    "\tfmt.Printf(\"%s：%d ms\\n\", name, time.Since(start).Milliseconds())\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用 mmap 随机读取文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mmap 随机读文件：1107 ms\n"
     ]
    }
   ],
   "source": [
    "package main\n",
    "\n",
    "import (\n",
    "\t\"fmt\"\n",
    "\t\"math/rand\"\n",
    "\t\"time\"\n",
    "\n",
    "\t\"golang.org/x/exp/mmap\"\n",
    ")\n",
    "\n",
    "const (\n",
    "\tmmapFile         = \"./23/random_sample.data\" // 与上一示例共用\n",
    "\trandomReadTimes  = 50000\n",
    ")\n",
    "\n",
    "func main() {\n",
    "\t// 使用 mmap.Open 将文件映射为内存区域\n",
    "\treader, err := mmap.Open(mmapFile)\n",
    "\tif err != nil {\n",
    "\t\tpanic(err)\n",
    "\t}\n",
    "\tdefer reader.Close()\n",
    "\n",
    "\tlength := reader.Len()\n",
    "\tif length == 0 {\n",
    "\t\tpanic(\"文件大小为 0，无法测试\")\n",
    "\t}\n",
    "\n",
    "\tbuffer := make([]byte, 1)\n",
    "\n",
    "\tmeasure(\"mmap 随机读文件\", func() {\n",
    "\t\tfor i := 0; i < randomReadTimes; i++ {\n",
    "\t\t\toffset := rand.Intn(length)\n",
    "\t\t\tif _, err := reader.ReadAt(buffer, int64(offset)); err != nil {\n",
    "\t\t\t\tpanic(err)\n",
    "\t\t\t}\n",
    "\t\t}\n",
    "\t})\n",
    "}\n",
    "\n",
    "func measure(name string, fn func()) {\n",
    "\tstart := time.Now()\n",
    "\tfn()\n",
    "\tfmt.Printf(\"%s：%d ms\\n\", name, time.Since(start).Milliseconds())\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 全量加载内存后随机访问"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "读取文件到内存：854 ms\n",
      "内存随机读（已载入）：0 ms\n",
      "\n",
      "总结：\n",
      "读取耗时：854 ms\n",
      "随机访问耗时：0 ms\n"
     ]
    }
   ],
   "source": [
    "package main\n",
    "\n",
    "import (\n",
    "\t\"fmt\"\n",
    "\t\"math/rand\"\n",
    "\t\"os\"\n",
    "\t\"time\"\n",
    ")\n",
    "\n",
    "const (\n",
    "\tloadFile        = \"./23/random_sample.data\" // 待读取的文件\n",
    "\trandomReadTimes = 50000                  // 随机读取次数\n",
    ")\n",
    "\n",
    "func main() {\n",
    "\tvar (\n",
    "\t\tdata []byte\n",
    "\t\terr  error\n",
    "\t)\n",
    "\n",
    "\t// 统计读取整个文件到内存所耗时间\n",
    "\tloadDuration := measure(\"读取文件到内存\", func() {\n",
    "\t\tdata, err = os.ReadFile(loadFile)\n",
    "\t\tif err != nil {\n",
    "\t\t\tpanic(err)\n",
    "\t\t}\n",
    "\t})\n",
    "\n",
    "\tlength := len(data)\n",
    "\tif length == 0 {\n",
    "\t\tpanic(\"文件大小为 0，无法测试\")\n",
    "\t}\n",
    "\n",
    "\t// 统计内存随机访问所耗时间\n",
    "\trandomDuration := measure(\"内存随机读（已载入）\", func() {\n",
    "\t\tvar sum byte\n",
    "\t\tfor i := 0; i < randomReadTimes; i++ {\n",
    "\t\t\toffset := rand.Intn(length)\n",
    "\t\t\tsum += data[offset]\n",
    "\t\t}\n",
    "\t\t_ = sum // 防止被编译器优化掉\n",
    "\t})\n",
    "\n",
    "\tfmt.Printf(\"\\n总结：\\n读取耗时：%d ms\\n随机访问耗时：%d ms\\n\",\n",
    "\t\tloadDuration.Milliseconds(), randomDuration.Milliseconds())\n",
    "}\n",
    "\n",
    "// measure 负责统一计时与打印\n",
    "func measure(name string, fn func()) time.Duration {\n",
    "\tstart := time.Now()\n",
    "\tfn()\n",
    "\telapsed := time.Since(start)\n",
    "\tfmt.Printf(\"%s：%d ms\\n\", name, elapsed.Milliseconds())\n",
    "\treturn elapsed\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Go (gonb)",
   "language": "go",
   "name": "gonb"
  },
  "language_info": {
   "codemirror_mode": "",
   "file_extension": ".go",
   "mimetype": "text/x-go",
   "name": "go",
   "nbconvert_exporter": "",
   "pygments_lexer": "",
   "version": "go1.24.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
